But one does give us an unbiased estimate of the ATT.

Why?

• Let’s think about what we’re estimating: the ATT.
• This is a weighted avearage of column means.

\[ \begin{aligned} \text{ATT} &= \frac{1}{m_1}\sum_{j:w_j=1} \qty{ \mu(1,x_j) - \mu(0,x_j)} \qfor \mu(w,x) = \frac{\sum_{j: w_j,x_j=w,x } y_j}{\sum_{j: w_j,x_j=w} 1} \qand m_w = \sum_{j: w_j=w} 1 \\ &= \sum_{x} \textcolor[RGB]{0,191,196}{p(1,x)} \qty{ \textcolor[RGB]{0,191,196}{\mu(1,x)} - \textcolor[RGB]{248,118,109}{\mu(0,x)}} \qfor p(w,x) = \frac{\sum_{j: w_j=w, x_j=x} 1}{\sum_{j: w_j=w} 1 } \\ &= \sum_{w,x} \alpha(w,x) \mu(w,x) \qfor \alpha(w,x) = \begin{cases}+p(1,x) & \text{ if } w=1 \\ -p(1,x) & \text{ if } w=0 \end{cases} \end{aligned} \]

• What you may notice is that, because these proportions \(p(w,x)\) are piecewise-constant, they’re in the model.

Claim

• You’ll get an unbiased estimate of the summary \(\sum_{w,x}\alpha(w,x) \mu(w,x)\)
• if the regression model \(\mathcal{M}\) contains at least one of …
  • the subpopulation mean function \(\mu(w,x)\)
  • the weights \(\alpha(w,x)\)

Prove it. \[ \textbf{Hint.} \qquad \frac{1}{n}\sum_{i=1}^n \qty{Y_i - \hat\mu(W_i,X_i)} m(W_i,X_i) = 0 \qqtext{ for every } m \in \mathcal{M} \]

Weighted Least Squares

• This sort of feels like an ‘a stopped clock is right twice a day’ thing.
• We chose a ‘bad model’ for \(\mu\) but it worked because it was a good model for \(\alpha\).
• But we can make this more systematic by using weighted least squares.

\[ \hat \mu = \mathop{\mathrm{argmin}}_{m \in \mathcal{M}} \sum_{i=1}^n \gamma(W_i,X_i) \qty{Y_i - m(W_i,X_i)}^2 \quad \text{ satisfies } \quad \sum_{i=1}^n \gamma(W_i,X_i) \qty{Y_i - \hat\mu(W_i,X_i)} m(W_i,X_i) = 0 \qqtext{ for every } m \in \mathcal{M} \]

• Our weights should reflect our summary.
• e.g. if we take \(\gamma(w,x) = \abs{\alpha(w,x)}\), we’re good if our model contains \(m(w,x) = \text{sign}(\alpha(w,x))\).
• For treatment effects, that means — if we choose the right weights — we just need to have indicators for treatment.

Inverse Probability Weighting

• This just sort of worked out for us ‘by chance’ with our block-randomized experiment.
• But let’s think about how to get the right weights in general. What are we doing?
• We’re averaging the the red and green curves over the distribution of the green points.

\[ \begin{aligned} \text{ATT} = \sum_{x} \textcolor[RGB]{0,191,196}{p(1,x) \hat\mu(w,x)} - \sum_{x} \textcolor[RGB]{0,191,196}{p(0,x)} \textcolor[RGB]{248,118,109}{\hat\mu(w,x)} \\ \end{aligned} \]

Does the First Term Cause Any Trouble?

\[ \begin{aligned} \text{ATT} = \sum_{x} \textcolor[RGB]{0,191,196}{p(1,x) \hat\mu(w,x)} - \sum_{x} \textcolor[RGB]{0,191,196}{p(0,x)} \textcolor[RGB]{248,118,109}{\hat\mu(w,x)} \\ \end{aligned} \]

• If our model has treatment-group indicators, no.

\[ \begin{aligned} 0 &= \mathop{\mathrm{E}}\qty[ \frac{1}{n}\sum_{i=1}^n \qty{Y_i - \hat\mu(W_i,X_i)} 1(W_i=1) ] \\ &=\mathop{\mathrm{E}}\qty[ \qty{\mu(W_i,X_i) - \hat\mu(W_i,X_i)} 1(W_i=1) ] \\ &=\mathop{\mathrm{E}}\qty[ \qty{\mu(1,X_i) - \hat\mu(1,X_i)} 1(W_i=1) ] \\ &=\mathop{\mathrm{E}}\qty[ \qty{\mu(1,X_i) - \hat\mu(1,X_i)} \mid W_i=1 ] P(W_i=1) \\ &= p_1 \sum_{x} \textcolor[RGB]{0,191,196}{p(1,x)} \qty{\mu(1,x) - \hat\mu(1,x)} \end{aligned} \]

• Intuitively, this is because we’re ‘fitting the data in the right place’
• We’re averaging over the same distribution of points we’ve fit our curve to.
• So, even if our model is ‘bad’ most places, its average is still right.

What About the Second?

\[ \begin{aligned} \text{ATT} = \sum_{x} \textcolor[RGB]{0,191,196}{p(1,x) \hat\mu(w,x)} - \sum_{x} \textcolor[RGB]{0,191,196}{p(0,x)} \textcolor[RGB]{248,118,109}{\hat\mu(w,x)} \\ \end{aligned} \]

• Even when our model has treatment-group indicators, it’s still trouble. Let’s try what we just did.

\[ \begin{aligned} 0 &= \mathop{\mathrm{E}}\qty[ \frac{1}{n}\sum_{i=1}^n \qty{Y_i - \hat\mu(W_i,X_i)} 1(W_i=0) ] \\ &=\mathop{\mathrm{E}}\qty[ \qty{\mu(W_i,X_i) - \hat\mu(W_i,X_i)} 1(W_i=0) ] \\ &=\mathop{\mathrm{E}}\qty[ \qty{\mu(0,X_i) - \hat\mu(0,X_i)} 1(W_i=1) ] \\ &=\mathop{\mathrm{E}}\qty[ \qty{\mu(,X_i) - \hat\mu(0,X_i)} \mid W_i=0 ] P(W_i=0) \\ &= p_0 \sum_{x} \textcolor[RGB]{248,118,109}{p(0,x)} \qty{\mu(0,x) - \hat\mu(0,x)} \end{aligned} \]

• This is not the right average. It’s the one that occurs in the ‘unadjusted gap’, not the ATT.
• We need to adjust for the fact that we’re averaging over a different distribution of points.

The Second: A Weighted Version

\[ \begin{aligned} \text{ATT} = \sum_{x} \textcolor[RGB]{0,191,196}{p(1,x) \hat\mu(w,x)} - \sum_{x} \textcolor[RGB]{0,191,196}{p(0,x)} \textcolor[RGB]{248,118,109}{\hat\mu(w,x)} \\ \end{aligned} \]

• Recall what we know about weighted least squares. \[ \hat \mu = \mathop{\mathrm{argmin}}_{m \in \mathcal{M}} \sum_{i=1}^n \gamma(W_i,X_i) \qty{Y_i - m(W_i,X_i)}^2 \quad \text{ satisfies } \quad \sum_{i=1}^n \gamma(W_i,X_i) \qty{Y_i - \hat\mu(W_i,X_i)} m(W_i,X_i) = 0 \qqtext{ for every } m \in \mathcal{M} \]

• Let’s try ‘upweighting’ the red points on the right side by the ratio of the number of green points to red ones.

\[ \begin{aligned} 0 &= \mathop{\mathrm{E}}\qty[ \frac{1}{n}\sum_{i=1}^n \gamma(X_i) \qty{Y_i - \hat\mu(W_i,X_i)} 1(W_i=0) ] \qfor \gamma(x)= \frac{\textcolor[RGB]{0,191,196}{p(1,x)}}{\textcolor[RGB]{248,118,109}{p(0,x)}} \\ &=\mathop{\mathrm{E}}\qty[ \gamma(X_i)\qty{\mu(W_i,X_i) - \hat\mu(W_i,X_i)} 1(W_i=0) ] \\ &=\mathop{\mathrm{E}}\qty[ \gamma(X_i)\qty{\mu(0,X_i) - \hat\mu(0,X_i)} 1(W_i=1) ] \\ &=\mathop{\mathrm{E}}\qty[ \gamma(X_i)\qty{\mu(,X_i) - \hat\mu(0,X_i)} \mid W_i=0 ] P(W_i=0) \\ &= p_0 \sum_{x} \qty{\gamma(x) \textcolor[RGB]{248,118,109}{p(0,x)}} \qty{\mu(0,x) - \hat\mu(0,x)} \\ &= p_0 \sum_{x} \qty{\frac{\textcolor[RGB]{0,191,196}{p(1,x)}}{\textcolor[RGB]{248,118,109}{p(0,x)}} \textcolor[RGB]{248,118,109}{p(0,x)}} \qty{\mu(0,x) - \hat\mu(0,x)} \\ &= p_0 \sum_{x} \textcolor[RGB]{0,191,196}{p(1,x)} \qty{\mu(0,x) - \hat\mu(0,x)} \end{aligned} \]

Summary

\[ \begin{aligned} \text{ATT} = \sum_{x} \textcolor[RGB]{0,191,196}{p(1,x) \hat\mu(1,x)} - \sum_{x} \textcolor[RGB]{0,191,196}{p(0,x) \hat\mu(0,x)} \qfor \hat \mu = \mathop{\mathrm{argmin}}_{m \in \mathcal{M}} \sum_{i=1}^n \gamma(W_i,X_i) \qty{Y_i - m(W_i,X_i)}^2 \qfor \frac{\textcolor[RGB]{0,191,196}{p(1,x)}}{p(w,x)} \end{aligned} \]

• This is the ‘inverse probability weighted least squares’ estimate of the ATT.
• We give equal weight to the green points because that’s the distribution we’re averaging over.
• But we weight the red points by the ratio of the number of green points to red ones so they look like they have that distribution too.

Summary

\[ \begin{aligned} \text{ATT} = \sum_{x} \textcolor[RGB]{0,191,196}{p(1,x) \hat\mu(1,x)} - \sum_{x} \textcolor[RGB]{0,191,196}{p(0,x)} \textcolor[RGB]{248,118,109}{\hat\mu(0,x)} \qfor \hat \mu = \mathop{\mathrm{argmin}}_{m \in \mathcal{M}} \sum_{i=1}^n \gamma(W_i,X_i) \qty{Y_i - m(W_i,X_i)}^2 \qfor \gamma(w,x) = \frac{\textcolor[RGB]{0,191,196}{p(1,x)}}{p(w,x)} \end{aligned} \]

• This is the ‘inverse probability weighted least squares’ estimate of the ATT.
• We give equal weight to the green points because that’s the distribution we’re averaging over.
• But we weight the red points by the ratio of the number of green points to red ones so they look like they have that distribution too.

   true unweighted weighted
1 0.244      0.000    0.000
2 0.244     -0.056    0.250
3 0.244      0.167    0.248
4 0.244      0.163    0.251
5 0.244      0.245    0.248
6 0.244      0.244    0.249

This Works For Any Distribution of Assignments

\[ \begin{aligned} \text{ATT} = \sum_{x} \textcolor[RGB]{0,191,196}{p(1,x) \hat\mu(1,x)} - \sum_{x} \textcolor[RGB]{0,191,196}{p(0,x)} \textcolor[RGB]{248,118,109}{\hat\mu(0,x)} \qfor \hat \mu = \mathop{\mathrm{argmin}}_{m \in \mathcal{M}} \sum_{i=1}^n \gamma(W_i,X_i) \qty{Y_i - m(W_i,X_i)}^2 \qfor \gamma(w,x) = \frac{\textcolor[RGB]{0,191,196}{p(1,x)}}{p(w,x)} \end{aligned} \]

• This is the ‘inverse probability weighted least squares’ estimate of the ATT.
• We give equal weight to the green points because that’s the distribution we’re averaging over.
• But we weight the red points by the ratio of the number of green points to red ones so they look like they have that distribution too.

   true unweighted weighted
1 0.233      0.000    0.000
2 0.233     -0.033    0.250
3 0.233      0.102    0.248
4 0.233      0.158    0.251
5 0.233      0.156    0.221
6 0.233      0.234    0.239

Even Weird Ones

\[ \begin{aligned} \text{ATT} = \sum_{x} \textcolor[RGB]{0,191,196}{p(1,x) \hat\mu(1,x)} - \sum_{x} \textcolor[RGB]{0,191,196}{p(0,x)} \textcolor[RGB]{248,118,109}{\hat\mu(0,x)} \qfor \hat \mu = \mathop{\mathrm{argmin}}_{m \in \mathcal{M}} \sum_{i=1}^n \gamma(W_i,X_i) \qty{Y_i - m(W_i,X_i)}^2 \qfor \gamma(w,x) = \frac{\textcolor[RGB]{0,191,196}{p(1,x)}}{p(w,x)} \end{aligned} \]

• This is the ‘inverse probability weighted least squares’ estimate of the ATT.
• We give equal weight to the green points because that’s the distribution we’re averaging over.
• But we weight the red points by the ratio of the number of green points to red ones so they look like they have that distribution too.

   true unweighted weighted
1 0.163      0.000    0.000
2 0.163      0.188    0.250
3 0.163      0.172    0.248
4 0.163      0.169    0.251
5 0.163      0.167    0.165
6 0.163      0.163    0.160